Phase One:
Scraping the data from one page:
Code Editor:
'''
We need four imports for this web-scrape.
'''
import pandas as pd #data science library
from bs4 import BeautifulSoup, Comment #HTML parser library
import requests #sleek HTTP library to interact with web-pages
import re #Regex library in Python
'''
The first thing we do here is request the web-page with the requests library.
This scrapes HTTP data from the source.
'''
page = requests.get('https://www.pro-football-reference.com/years/2000/')
'''
Next we place all the page data into a bs4 object. Which is used to extract the
HTML content from the page.
'''
soup = BeautifulSoup(page.content, 'lxml')
'''
The bs4 object contains tables within. We can extract these tables
into a list with a method inherent to bs4. This method finds the start and end
tags for each table in the soup object and appends them into a list within a bs4
object.
'''
tables = soup.find_all('table')
'''
So, if we print tables[0] we will see the first table in HTML format. We can now
convert it to a dataframe with Pandas.
'''
afc_standings_df = pd.read_html(str(tables))[0]
'''
We can do the same for the second table.
'''
nfc_standings_df = pd.read_html(str(tables))[1]
Python Console:
'''
We have now extracted and can view the first two dataframes.
'''
In[]: afc_standings_df.head()
Out[]:
Tm W L W-L% PF PA \
0 AFC East AFC East AFC East AFC East AFC East AFC East
1 Miami Dolphins* 11 5 .688 323 226
2 Indianapolis Colts+ 10 6 .625 429 326
3 New York Jets 9 7 .563 321 321
4 Buffalo Bills 8 8 .500 315 350
PD MoV SoS SRS OSRS DSRS
0 AFC East AFC East AFC East AFC East AFC East AFC East
1 97 6.1 1.0 7.1 0.0 7.1
2 103 6.4 1.5 7.9 7.1 0.8
3 0 0.0 3.5 3.5 1.4 2.2
4 -35 -2.2 2.2 0.0 0.5 -0.5
In[]: nfc_standings_df.head()
Out[]:
Tm W L W-L% PF PA \
0 NFC East NFC East NFC East NFC East NFC East NFC East
1 New York Giants* 12 4 .750 328 246
2 Philadelphia Eagles+ 11 5 .688 351 245
3 Washington Redskins 8 8 .500 281 269
4 Dallas Cowboys 5 11 .313 294 361
PD MoV SoS SRS OSRS DSRS
0 NFC East NFC East NFC East NFC East NFC East NFC East
1 82 5.1 -2.7 2.4 -1.3 3.8
2 106 6.6 -3.6 3.1 1.0 2.1
3 12 0.8 0.2 1.0 -2.9 3.8
4 -67 -4.2 -0.4 -4.6 -1.5 -3.0
Code Editor:
'''
Now let's scrape the Team Offense dataframe. To do this start by extracting all
the comments in the bs4 object.
'''
comments = soup.findAll(text=lambda text:isinstance(text, Comment))
'''
Next we create a regex object to specify a table id.
This Regex is looking for an open table tag(<table), which matches any character
except a new line which matches zero or more repitions (.+?), of the named group
(id="team_stats"), where the potential match dose or does not contain a white space
character ([\s\S]), and ends with a table end tag (</table>).
'''
rx = re.compile(r'<table.+?id="team_stats".+?>[\s\S]+?</table>')
'''
Then use a loop to iterate through each list item in the comments object with
the Regex pattern.
Then, once the Regex pattern is matched it breaks out of the loop.
'''
for c in comments:
try:
table = rx.search(c.string).group(0)
# break the loop if found
break
except:
pass
'''
The table is in HTML format. Convert it with Pandas. The output is in a list
so call it by it's index—here it is 0.
'''
df_team_stats = pd.read_html(str(table))[0]
'''
Now let's get Passing Offense using the same methods.
'''
rx = re.compile(r'<table.+?id="passing".+?>[\s\S]+?</table>')
for c in comments:
try:
table = rx.search(c.string).group(0)
# break the loop if found
break
except:
pass
df_passing_stats = pd.read_html(str(table))[0]
Python Console:
In[]: df_team_stats.head()
Out[]:
Unnamed: 0_level_0 Unnamed: 1_level_0 Unnamed: 2_level_0 \
Rk Tm G
0 1.0 St. Louis Rams 16.0
1 2.0 Denver Broncos 16.0
2 3.0 Oakland Raiders 16.0
3 4.0 Indianapolis Colts 16.0
4 5.0 Minnesota Vikings 16.0
Unnamed: 3_level_0 Unnamed: 4_level_0 Tot Yds & TO \
PF Yds Ply Y/P TO
0 540.0 7075.0 1014.0 7.0 35.0
1 485.0 6554.0 1115.0 5.9 25.0
2 479.0 5776.0 1023.0 5.6 20.0
3 429.0 6141.0 1026.0 6.0 29.0
4 397.0 5961.0 958.0 6.2 28.0
Unnamed: 8_level_0 Unnamed: 9_level_0 Passing \
FL 1stD Cmp Att Yds TD Int
0 12.0 380.0 380.0 587.0 5232.0 37.0 23.0
1 13.0 383.0 354.0 569.0 4243.0 28.0 12.0
2 9.0 337.0 284.0 475.0 3306.0 28.0 11.0
3 14.0 357.0 357.0 571.0 4282.0 33.0 15.0
4 10.0 319.0 307.0 495.0 3832.0 33.0 18.0
Rushing Penalties \
NY/A 1stD Att Yds TD Y/A 1stD Pen Yds 1stPy
0 8.3 247.0 383.0 1843.0 26.0 4.8 112.0 111.0 942.0 21.0
1 7.1 223.0 516.0 2311.0 21.0 4.5 124.0 89.0 792.0 36.0
2 6.6 177.0 520.0 2470.0 23.0 4.8 128.0 118.0 940.0 32.0
3 7.2 213.0 435.0 1859.0 14.0 4.3 111.0 89.0 866.0 33.0
4 7.2 193.0 428.0 2129.0 14.0 5.0 107.0 106.0 908.0 19.0
Unnamed: 25_level_0 Unnamed: 26_level_0 Unnamed: 27_level_0
Sc% TO% EXP
0 46.8 16.7 49.41
1 38.3 11.7 -40.21
2 39.2 10.6 11.03
3 40.4 15.2 47.85
4 39.7 14.9 -60.12
In[]: df_passing_stats.head()
Out[]:
Rk Tm G Cmp Att Cmp% Yds TD TD% \
0 1.0 St. Louis Rams 16.0 380.0 587.0 64.7 5232.0 37.0 6.3
1 2.0 Indianapolis Colts 16.0 357.0 571.0 62.5 4282.0 33.0 5.8
2 3.0 Denver Broncos 16.0 354.0 569.0 62.2 4243.0 28.0 4.9
3 4.0 San Francisco 49ers 16.0 366.0 583.0 62.8 4239.0 32.0 5.5
4 5.0 Kansas City Chiefs 16.0 342.0 582.0 58.8 4149.0 29.0 5.0
Int Int% Lng Y/A AY/A Y/C Y/G Rate Sk Yds.1 NY/A ANY/A \
0 23.0 3.9 85.0 9.4 8.9 14.5 327.0 99.7 44.0 260.0 8.3 7.8
1 15.0 2.6 78.0 7.7 7.7 12.4 267.6 94.7 20.0 131.0 7.2 7.2
2 12.0 2.1 61.0 7.8 7.9 12.6 265.2 94.2 30.0 221.0 7.1 7.1
3 10.0 1.7 69.0 7.5 7.9 12.0 264.9 97.0 25.0 161.0 7.0 7.3
4 15.0 2.6 81.0 7.6 7.4 12.9 259.3 88.5 34.0 259.0 6.7 6.6
Sk% 4QC GWD EXP
0 7.0 1.0 3.0 120.25
1 3.4 2.0 3.0 88.16
2 5.0 2.0 3.0 37.99
3 4.1 NaN NaN 53.79
4 5.5 3.0 3.0 36.06
'''
Here notice there are unnamed column headers in df_team_stats. This is becuase
it is multi-indexed. Meaning there are headers in the first index that describe
headers in the second. Here's what they look like when you call the columns.
'''
In[]: df_team_stats.columns
Out[]:
MultiIndex([( 'Unnamed: 0_level_0', 'Rk'),
( 'Unnamed: 1_level_0', 'Tm'),
( 'Unnamed: 2_level_0', 'G'),
( 'Unnamed: 3_level_0', 'PF'),
( 'Unnamed: 4_level_0', 'Yds'),
( 'Tot Yds & TO', 'Ply'),
( 'Tot Yds & TO', 'Y/P'),
( 'Tot Yds & TO', 'TO'),
( 'Unnamed: 8_level_0', 'FL'),
( 'Unnamed: 9_level_0', '1stD'),
( 'Passing', 'Cmp'),
( 'Passing', 'Att'),
( 'Passing', 'Yds'),
( 'Passing', 'TD'),
( 'Passing', 'Int'),
( 'Passing', 'NY/A'),
( 'Passing', '1stD'),
( 'Rushing', 'Att'),
( 'Rushing', 'Yds'),
( 'Rushing', 'TD'),
( 'Rushing', 'Y/A'),
( 'Rushing', '1stD'),
( 'Penalties', 'Pen'),
( 'Penalties', 'Yds'),
( 'Penalties', '1stPy'),
('Unnamed: 25_level_0', 'Sc%'),
('Unnamed: 26_level_0', 'TO%'),
('Unnamed: 27_level_0', 'EXP')],
)
'''
We can combine them to make this easier to work with and read. The following
takes every item in the first index and joins it with a "_" to every item
in the second index.
'''
In[]: df_team_stats.columns = ['_'.join(col) for col in df_team_stats.columns]
In[]: df_team_stats.columns
Out[]:
Index(['Unnamed: 0_level_0_Rk', 'Unnamed: 1_level_0_Tm',
'Unnamed: 2_level_0_G', 'Unnamed: 3_level_0_PF',
'Unnamed: 4_level_0_Yds', 'Tot Yds & TO_Ply', 'Tot Yds & TO_Y/P',
'Tot Yds & TO_TO', 'Unnamed: 8_level_0_FL', 'Unnamed: 9_level_0_1stD',
'Passing_Cmp', 'Passing_Att', 'Passing_Yds', 'Passing_TD',
'Passing_Int', 'Passing_NY/A', 'Passing_1stD', 'Rushing_Att',
'Rushing_Yds', 'Rushing_TD', 'Rushing_Y/A', 'Rushing_1stD',
'Penalties_Pen', 'Penalties_Yds', 'Penalties_1stPy',
'Unnamed: 25_level_0_Sc%', 'Unnamed: 26_level_0_TO%',
'Unnamed: 27_level_0_EXP'],
dtype='object')
'''
But we're not done fixing the columns. We have to get rid of the Unnamed:
portions of index 1 that joined to index 2. We can do that by using Regex.
So first we create a new Regex pattern to match the unwanted string data
in the index.
So, we're trying to match the following where
n=someDigit: "Unnamed: ([n]|[n][n])_level_n_"
Be aware that regex is strict.
'''
#The below means:
#match "Unnamed: ([a number 0-9]or[a numner 0-5]followed by[a number 0-9]_level_\anyDigitFrom0-9_)"
In[]: re_col_pat = r'(Unnamed: ([0-9]|[0-5][0-9])_level_\d_)'
'''
Now we iterate through every column header in our columns and replace any exact
matches to re_col_pat with nothing and append it to a new list of column headers.
'''
In[]: new_cols = []
for i in df_team_stats.columns:
res = re.sub(re_col_pat , '', i)
new_cols.append(res)
'''
Now we replace the old columns with new ones by putting them both into a
dictionary and renaming them with that dictionary. The rename() method in
Pandas looks like someDf.rename(old_col, new_col). You can also use
this method with a dictionary to rename multiple columns like
someDf.rename({old_col_1:new_col_1, old_col_2:new_col_2}).
'''
In[]: new_col_dict = dict(zip(df_team_stats.columns,new_cols))
In[]: df_team_stats.rename(columns=new_col_dict, inplace=True)
In[]: df_team_stats.columns
Out[]:
Index(['Rk', 'Tm', 'G', 'PF', 'Yds', 'Tot Yds & TO_Ply', 'Tot Yds & TO_Y/P',
'Tot Yds & TO_TO', 'FL', '1stD', 'Passing_Cmp', 'Passing_Att',
'Passing_Yds', 'Passing_TD', 'Passing_Int', 'Passing_NY/A',
'Passing_1stD', 'Rushing_Att', 'Rushing_Yds', 'Rushing_TD',
'Rushing_Y/A', 'Rushing_1stD', 'Penalties_Pen', 'Penalties_Yds',
'Penalties_1stPy', 'Sc%', 'TO%', 'EXP'],
dtype='object')
'''
Now we'll want to insert a year column because eventually we're going to get all
the data from 2000-2020. This data is for 2000 we may want to be able to
differentiate it from other years.
'''
In[]: df_team_stats.insert(loc=0, column='Yr', value=2000)
In[]: df_team_stats.head()
Out[]:
Yr Rk Tm G PF Yds Tot Yds & TO_Ply \
0 2000 1.0 St. Louis Rams 16.0 540.0 7075.0 1014.0
1 2000 2.0 Denver Broncos 16.0 485.0 6554.0 1115.0
2 2000 3.0 Oakland Raiders 16.0 479.0 5776.0 1023.0
3 2000 4.0 Indianapolis Colts 16.0 429.0 6141.0 1026.0
4 2000 5.0 Minnesota Vikings 16.0 397.0 5961.0 958.0
Tot Yds & TO_Y/P Tot Yds & TO_TO FL 1stD Passing_Cmp Passing_Att \
0 7.0 35.0 12.0 380.0 380.0 587.0
1 5.9 25.0 13.0 383.0 354.0 569.0
2 5.6 20.0 9.0 337.0 284.0 475.0
3 6.0 29.0 14.0 357.0 357.0 571.0
4 6.2 28.0 10.0 319.0 307.0 495.0
Passing_Yds Passing_TD Passing_Int Passing_NY/A Passing_1stD \
0 5232.0 37.0 23.0 8.3 247.0
1 4243.0 28.0 12.0 7.1 223.0
2 3306.0 28.0 11.0 6.6 177.0
3 4282.0 33.0 15.0 7.2 213.0
4 3832.0 33.0 18.0 7.2 193.0
Rushing_Att Rushing_Yds Rushing_TD Rushing_Y/A Rushing_1stD \
0 383.0 1843.0 26.0 4.8 112.0
1 516.0 2311.0 21.0 4.5 124.0
2 520.0 2470.0 23.0 4.8 128.0
3 435.0 1859.0 14.0 4.3 111.0
4 428.0 2129.0 14.0 5.0 107.0
Penalties_Pen Penalties_Yds Penalties_1stPy Sc% TO% EXP
0 111.0 942.0 21.0 46.8 16.7 49.41
1 89.0 792.0 36.0 38.3 11.7 -40.21
2 118.0 940.0 32.0 39.2 10.6 11.03
3 89.0 866.0 33.0 40.4 15.2 47.85
4 106.0 908.0 19.0 39.7 14.9 -60.12
'''
There we have a working dataframe for offensive team stats for the year 2000.
Let's move on to get all the dataframes on the page without having to rewrite the
code for every one.
'''
Phase Two:
And here's the passing offense table id:
I've already gathered all the table id's for each dataframe. So now let's write our function to scrape all the dataframes on the one page—for year 2000.
I'm also going to add a dictionary that we will use to make the team names uniform across all the years. So, the team either moved to another city/state or they just changed the current name. We need the team names to be stanardized. I'm going to do it right when we scrape the data. From 2000 to 2020 there have been 4 team name changes.
Code Editor
#LET'S WRITE THE FUNCTION TO TAKE 1 ARGUMENT UNIVERSALLY DIFFERENT TO EACH
#HTML PAGE HOUSING THE DATA; THE URL AND THE YEAR.
def get_all_data(url, yr):
#NEXT DO ALMOST EVERYTHING WE'VE ALREADY DONE.
r = requests.get(url)
soup = BeautifulSoup(r.text, 'lxml')
#REG EX PATTERN FOR REMAPPING COLS
re_col_pat = r'(Unnamed: ([0-9]|[0-5][0-9])_level_\d_)'
#TEAM NAME CHANGE DICTIONARY
team_name_change_dict = {
'St. Louis Rams':'Los Angeles Rams',
'Oakland Raiders':'Las Vegas Raiders',
'San Diego Chargers':'Los Angeles Chargers',
'Washington Redskins':'Washington Football Team',
}
#CONVERT AVAILIBLE AFC/NFC TABLEs (NOT HIDDEN IN COMMENTS)
tables = soup.find_all('table') #standings table
#ITERATE THROUGH TABLES AND SCRAPE AFC/NFC STANDINGS; THE LOOP ITERATES
#TWICE, ONE FOR EACH TABLE IN TABLES
for i3 in range(len(tables)):
df_s = pd.read_html(str(tables))[i3]
df_s.columns = ['_'.join(col) for col in df_s.columns]
#NEW COL NAMES
new_cols = []
for i4 in df_s.columns:
res = re.sub(re_col_pat, '', i4)
new_cols.append(res)
#CREATE NEW COLS/OLD COLS DICT
new_col_dict = dict(zip(df_s.columns,new_cols))
#APPLY NEW COL NAMES WITH RENAME()
df_s.rename(columns=new_col_dict, inplace=True)
df_s.rename(columns={'T_m':'Tm'}, inplace=True)
#CHANGE TEAM NAMES SO THEY ARE UNIFORM
df_s.replace(to_replace=team_name_change_dict, regex=True, inplace=True)
#INSERT YEAR COL
df_s.insert(loc=0, column='Yr', value=yr)
if i3 == 0:
df_afc_standings = df_s
elif i3 == 1:
df_nfc_standings = df_s
#NOW WE SCRAPE THE DATAFRAMES HIDDEN IN THE COMMENTS
#FIRST INSTANIATE A LIST WITH ALL THE ID'S OF THE HTML TABLES
df_str_lst = ['team_stats', 'passing', 'rushing',
'returns', 'kicking', 'team_scoring',
'team_conversions', 'drives', 'playoff_results']
#SCRAPE THE COMMENTS FROM THE PAGE SO WE CAN ITERATE THROUGH THEM AND GRAB THE DATAFRAMES
comments = soup.findAll(text=lambda text:isinstance(text, Comment))
#ITERATE THROUGH DF_STR_LST WITH A FOR LOOP AND FORMAT THE ITEM ITERATING INTO THE REGEX
for i in range(len(df_str_lst)):
rx_compiler_str = r'<table.+?id="{}".+?>[\s\S]+?</table>'.format(df_str_lst[i])
#SAME AS BEFORE INSTANTIATE A REGEX COMPILED OBJECT AND ITERATE THROUGH THE COMMENTS
rx = re.compile(rx_compiler_str)
for comment in comments:
try:
table = rx.search(comment.string).group(0)
# break the loop if found
break
except:
pass
#CONVERT TABLE AND CLEAN THE DF
df = pd.read_html(str(table))[0]
df.columns = ['_'.join(col) for col in df.columns]
#NEW COL NAMES
new_cols = []
for i2 in df.columns:
res = re.sub(re_col_pat , '', i2)
new_cols.append(res)
#CREATE NEW COLS/OLD COLS DICT
new_col_dict = dict(zip(df.columns,new_cols))
#APPLY NEW COL NAMES WITH RENAME()
df.rename(columns=new_col_dict, inplace=True)
df.rename(columns={'T_m':'Tm'}, inplace=True)
#UPDATE TEAM NAMES SO THEY ARE ALL UNIFORM
df.replace(to_replace=team_name_change_dict, regex=True, inplace=True)
#INSERT YEAR COL
df.insert(loc=0, column='Yr', value=yr)
#NOW USE AN IF STATEMENT THAT CHECKS WHICH TABLE ID THE ITERATION IS ON
#AND ASSIGN IT TO THE CORRESPONDING DATAFRAME
if i == 0:
df_team_stats = df
elif i == 1:
df_passing = df
elif i == 2:
df_rushing = df
elif i == 3:
df_returns = df
elif i == 4:
df_kicking = df
elif i == 5:
df_team_scoring = df
elif i == 6:
df_team_conversions = df
elif i == 7:
df_team_drives = df
elif i == 8:
df_playoff_results = df
#LASTLY RETURN ALL THE DATAFRAMES
return [df_afc_standings, df_nfc_standings,
df_team_stats, df_passing,
df_rushing, df_returns,
df_kicking, df_team_scoring,
df_team_conversions, df_team_drives,
df_playoff_results]
#call the function for the data; each item in the return list is reassigned to a
#variable dataframe
#I reassigned them with the addition of _mast to denote master dataframes for all years
[df_afc_standings_mast, df_nfc_standings_mast,
df_team_stats_mast, df_passing_mast,
df_rushing_mast, df_returns_mast,
df_kicking_mast, df_team_scoring_mast,
df_team_conversions_mast, df_team_drives_mast,
df_playoff_results_mast] = get_all_data('https://www.pro-football-reference.com/years/2000/', 2000)
'''
Now we have all the dataframes on the page.
'''
In[]: df_passing_mast.head()
Out[]:
Yr R_k T_m G C_m_p A_t_t C_m_p_% Y_d_s T_D \
0 2000 1.0 Los Angeles Rams 16.0 380.0 587.0 64.7 5232.0 37.0
1 2000 2.0 Indianapolis Colts 16.0 357.0 571.0 62.5 4282.0 33.0
2 2000 3.0 Denver Broncos 16.0 354.0 569.0 62.2 4243.0 28.0
3 2000 4.0 San Francisco 49ers 16.0 366.0 583.0 62.8 4239.0 32.0
4 2000 5.0 Kansas City Chiefs 16.0 342.0 582.0 58.8 4149.0 29.0
T_D_% I_n_t I_n_t_% L_n_g Y_/_A A_Y_/_A Y_/_C Y_/_G R_a_t_e S_k \
0 6.3 23.0 3.9 85.0 9.4 8.9 14.5 327.0 99.7 44.0
1 5.8 15.0 2.6 78.0 7.7 7.7 12.4 267.6 94.7 20.0
2 4.9 12.0 2.1 61.0 7.8 7.9 12.6 265.2 94.2 30.0
3 5.5 10.0 1.7 69.0 7.5 7.9 12.0 264.9 97.0 25.0
4 5.0 15.0 2.6 81.0 7.6 7.4 12.9 259.3 88.5 34.0
Y_d_s_._1 N_Y_/_A A_N_Y_/_A S_k_% 4_Q_C G_W_D E_X_P
0 260.0 8.3 7.8 7.0 1.0 3.0 120.25
1 131.0 7.2 7.2 3.4 2.0 3.0 88.16
2 221.0 7.1 7.1 5.0 2.0 3.0 37.99
3 161.0 7.0 7.3 4.1 NaN NaN 53.79
4 259.0 6.7 6.6 5.5 3.0 3.0 36.06
In[]: df_team_drives_mast.head()
Out[]:
Yr Rk Tm G Dr Plays Sc% TO% \
0 2000 1.0 Los Angeles Chargers 16.0 206 1007 21.4 23.3
1 2000 2.0 New York Jets 16.0 205 1109 27.8 19.0
2 2000 3.0 Detroit Lions 16.0 204 1034 26.5 14.2
3 2000 4.0 Buffalo Bills 16.0 200 1118 29.0 10.5
4 2000 5.0 Jacksonville Jaguars 16.0 198 1113 34.8 13.6
Average Drive_Plays Average Drive_Yds Average Drive_Start \
0 4.9 20.8 Own 29.3
1 5.4 26.3 Own 33.2
2 5.1 21.6 Own 35.9
3 5.6 27.4 Own 29.4
4 5.6 28.7 Own 34.1
Average Drive_Time Average Drive_Pts
0 2:12 1.15
1 2:23 1.50
2 2:22 1.38
3 2:35 1.51
4 2:33 1.85
Phase Three:
There are a few things we will need. We need a list of the urls we're scraping and a list of years to scrape.
Code Editor:
import pandas as pd
from bs4 import BeautifulSoup, Comment
import requests, re
pd.options.display.max_columns = 50
pd.options.display.max_rows = 50
pd.set_option('max_colwidth', 50)
pd.set_option('max_rows', 50)
def get_all_data(url_lst):
#IF/ELSE STATMENT CHECKS FOR A LIST OR STR
if isinstance(url_lst, list):
print('List is being processed...')
else:
url_lst = [url_lst]
print('processing...')
#REG EX PATTERN FOR REMAPPING COLS
re_col_pat = r'(Unnamed: ([0-9]|[0-5][0-9])_level_\d_)'
#REG EX PATTERN FOR GETTING THE YEAR FROM THE URL
re_year_pat = r'(\d\d\d\d)'
#TEAM NAME CHANGE DICTIONARY
team_name_change_dict = {
'St. Louis Rams':'Los Angeles Rams',
'Oakland Raiders':'Las Vegas Raiders',
'San Diego Chargers':'Los Angeles Chargers',
'Washington Redskins':'Washington Football Team',
}
#CREATE EMPTY MASTER DATAFRAMES AS PLACEHOLDERS
[df_afc_standings_mast, df_nfc_standings_mast,
df_team_stats_mast, df_passing_mast,
df_rushing_mast, df_returns_mast,
df_kicking_mast, df_team_scoring_mast,
df_team_conversions_mast, df_team_drives_mast,
df_playoff_results_mast] = [pd.DataFrame() for n in range(11)]
for n in url_lst:
#BS4 DATA PARSE
r = requests.get(n)
soup = BeautifulSoup(r.text, 'lxml')
#EXTRACT THE YEAR FROM THE URL STRING IN CURRENT ITERATION
yr = re.findall(re_year_pat, n)
#CONVERT AVAILIBLE AFC/NFC TABLEs (NOT HIDDEN IN COMMENTS)
tables = soup.find_all('table') #standings table
#####################################################################
#GRAB AFC/NFC STANDINGS
for i3 in range(len(tables)):
df_s = pd.read_html(str(tables))[i3]
df_s.columns = ['_'.join(col) for col in df_s.columns]
#NEW COL NAMES
new_cols = []
for i4 in df_s.columns:
res = re.sub(re_col_pat, '', i4)
new_cols.append(res)
#CREATE NEW COLS/OLD COLS DICT
new_col_dict = dict(zip(df_s.columns,new_cols))
#APPLY NEW COL NAMES WITH RENAME()
df_s.rename(columns=new_col_dict, inplace=True)
df_s.rename(columns={'T_m':'Tm'}, inplace=True)
#CHANGE TEAM NAMES SO THEY ARE UNIFORM
df_s.replace(to_replace=team_name_change_dict, regex=True, inplace=True)
#INSERT YEAR COL
df_s.insert(loc=0, column='Yr', value=yr[0])
if i3 == 0:
df_afc_standings = df_s
df_afc_standings_mast = df_afc_standings_mast.append(df_afc_standings)
elif i3 == 1:
df_nfc_standings = df_s
df_nfc_standings_mast = df_nfc_standings_mast.append(df_nfc_standings)
#####################################################################
#GRAB THE DFs HIDDEN IN COMMENTS
df_str_lst = ['team_stats', 'passing', 'rushing',
'returns', 'kicking', 'team_scoring',
'team_conversions', 'drives', 'playoff_results']
#GET THE COMMENTS
comments = soup.findAll(text=lambda text:isinstance(text, Comment))
#CREATE A MAST DF
for i in range(len(df_str_lst)):
rx_compiler_str = r'<table.+?id="{}".+?>[\s\S]+?</table>'.format(df_str_lst[i])
#LOOK FOR TABLE WITH THE ID "TEAM_STATS"
rx = re.compile(rx_compiler_str)
for comment in comments:
try:
table = rx.search(comment.string).group(0)
# break the loop if found
break
except:
pass
#CONVERT TABLE AND CLEAN THE DF COLS
df = pd.read_html(str(table))[0]
df.columns = ['_'.join(col) for col in df.columns]
#NEW COL NAMES
new_cols = []
for i2 in df.columns:
res = re.sub(re_col_pat , '', i2)
new_cols.append(res)
#CREATE NEW COLS/OLD COLS DICT
new_col_dict = dict(zip(df.columns,new_cols))
#APPLY NEW COL NAMES WITH RENAME()
df.rename(columns=new_col_dict, inplace=True)
df.rename(columns={'T_m':'Tm'}, inplace=True)
#CHANGE TEAM NAMES SO THEY ARE UNIFORM
df.replace(to_replace=team_name_change_dict, regex=True, inplace=True)
#INSERT YEAR COL
df.insert(loc=0, column='Yr', value=yr[0])
#RETURN THE CORRECT DF TO CORRESPONDING MASTER DATASET PER ITERATION
if i == 0:
df_team_stats = df
df_team_stats_mast = df_team_stats_mast.append(df_team_stats)
elif i == 1:
df_passing = df
df_passing_mast = df_passing_mast.append(df_passing)
elif i == 2:
df_rushing = df
df_rushing_mast = df_rushing_mast.append(df_rushing)
elif i == 3:
df_returns = df
df_returns_mast = df_returns_mast.append(df_returns)
elif i == 4:
df_kicking = df
df_kicking_mast = df_kicking_mast.append(df_kicking)
elif i == 5:
df_team_scoring = df
df_team_scoring_mast = df_team_scoring_mast.append(df_team_scoring)
elif i == 6:
df_team_conversions = df
df_team_conversions_mast = df_team_conversions_mast.append(df_team_conversions)
elif i == 7:
df_team_drives = df
df_team_drives_mast = df_team_drives_mast.append(df_team_drives)
elif i == 8:
df_playoff_results = df
df_playoff_results_mast = df_playoff_results_mast.append(df_playoff_results)
#RETURN THE MASTER DATAFRAMES
return [df_afc_standings_mast, df_nfc_standings_mast,
df_team_stats_mast, df_passing_mast,
df_rushing_mast, df_returns_mast,
df_kicking_mast, df_team_scoring_mast,
df_team_conversions_mast, df_team_drives_mast,
df_playoff_results_mast]
'''
Now because nobody wants to copy and paste 20 url's into a list we'll write them
a list comprehension of all the years 2000 to 2020 and a loop to format the year
into the link (the links are all the same except for the year). Then append it
to a stand-alone url list.
'''
year_lst = [i for i in range(2000,2021)]
link_lst = []
for i in range(len(year_lst)):
url = "http://www.pro-football-reference.com/years/{}/".format(year_lst[i])
link_lst.append(url)
del i
del url
'''
Now that we have all our links pass them into the function.
'''
[df_afc_standings_mast, df_nfc_standings_mast,
df_team_stats_mast, df_passing_mast,
df_rushing_mast, df_returns_mast,
df_kicking_mast, df_team_scoring_mast,
df_team_conversions_mast, df_team_drives_mast,
df_playoff_results_mast] = get_all_data(link_lst)